Papers with Visual storytelling
SCO-VIST: Social Interaction Commonsense Knowledge-based Visual Storytelling (2024.eacl-long)
Copied to clipboard
| Challenge: | Visual storytelling aims to automatically generate a coherent story based on a given image sequence. |
| Approach: | They propose a framework that represents the image sequence as a graph with objects and relations that includes human action motivation and its social interaction commonsense knowledge. |
| Outcome: | The proposed framework produces stories superior across multiple metrics in terms of visual grounding, coherence, diversity, and humanness, per both automatic and human evaluations. |
Keep it Consistent: Topic-Aware Storytelling from an Image Stream via Iterative Multi-agent Communication (2020.coling-main)
Copied to clipboard
| Challenge: | Existing methods for visual storytelling construct text description independently for each image and roughly concatenate them as a story, which leads to the problem of generating semantically incoherent content. |
| Approach: | They propose a topic description task to detect the global semantic context of an image stream and a story is then constructed with the guidance of the topic description. |
| Outcome: | The proposed framework can generate stories with higher quality compared to state-of-the-art methods on a VIST dataset. |
RoViST: Learning Robust Metrics for Visual Storytelling (2022.findings-naacl)
Copied to clipboard
| Challenge: | Visual storytelling is the task of generating a story paragraph that describes a given image sequence. |
| Approach: | They propose 3 evaluation metrics sets that analyze which aspects we would look for in a good story . they compare their correlation with human judgement scores on a sample of machine stories . |
| Outcome: | The proposed evaluation metrics outperform other metrics on human correlation on a sample of machine stories from state-of-the-art models. |
Visual Storytelling with Question-Answer Plans (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models focus on enhancing the representation of image sequences, but the stories are repetitive, illogical, and lacking in detail. |
| Approach: | They propose a framework which integrates visual representations with pretrained language models and planning. |
| Outcome: | The proposed framework combines visual representations with pretrained language models and planning. |
Learning to Rank Visual Stories From Human Ranking Data (2022.acl-long)
Copied to clipboard
| Challenge: | Existing studies on visual storytelling (VIST) use automated evaluation metrics for text generation. |
| Approach: | They develop a Vrank metric that repurposes human evaluation results for automatic evaluation. |
| Outcome: | The proposed model is more accurate than existing metrics and is generalizable to textual stories. |
Not (yet) the whole story: Evaluating Visual Storytelling Requires More than Measuring Coherence, Grounding, and Repetition (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Visual storytelling is a task of generating a story for a sequence of several temporally-ordered images or video frames. |
| Approach: | They propose a method that measures story quality in terms of human likeness regarding three key aspects highlighted in previous work: visual grounding, coherence, and repetitiveness. |
| Outcome: | The proposed method improves on the foundation model LLaVA but only slightly compared to TAPM, a 50-times smaller visual storytelling model. |